cient Layering for High Speed Communication : the MPI over Fast Messages ( FM )

We describe our experience designing, implementing, and evaluating two generations of high performance communication libraries, Fast Messages (FM) for Myrinet. In FM 1, we designed a simple interface and provided guarantees of reliable and in-order delivery, and ow control. While this was a signiicant improvement over previous systems, it was not enough. Layering MPI atop FM 1 showed that only about 35% of the FM 1 bandwidth could be delivered to higher level communication APIs. Our second generation communication layer, FM 2, addresses the identiied problems, providing gather-scatter, interlayer scheduling, receiver ow control, as well as some convenient API features which simplify programming. FM 2 can deliver 55-95% to higher level APIs such as MPI. This is especially impressive as the absolute bandwidths delivered have increased over fourfold to 90 MB/s. We describe general issues encountered in matching two communication layers, and our solutions as embodied in FM 2.